Traffic Monitoring and Reporting System and Method

ABSTRACT

A system and method for monitoring vehicle traffic and collecting data indicative of pedestrian right of way violations by vehicles is provided. The system comprises memory and logic for monitoring traffic intersections and recording evidence indicating that vehicles have violated pedestrian right of way. Two sensor modalities collecting video data and radar data of the intersection under observation are employed in one embodiment of the system. The violation evidence can be accessed remotely by a traffic official for issuing of traffic citations.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/244,712 entitled “Pedestrian Right of Way Monitoring and ReportingSystem and Method” and filed on Aug. 23, 2016, which is a continuationof U.S. application Ser. No. 14/257,472, entitled “Pedestrian Right ofWay Monitoring and Reporting System and Method” and filed on Apr. 21,2014, which claims the benefit of and priority to Provisional PatentApplication U.S. Ser. No. 61/813,783, entitled “Automotive System forEnforcement and Safety” and filed on Apr. 19, 2013. All of theseapplications are fully incorporated herein by reference.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under Contract NumberDTRT57-13-C-10004 awarded by the Department of Transportation, FederalHighway Administration. The government has certain rights in theinvention.

BACKGROUND AND SUMMARY

A system and method for monitoring vehicle traffic and reportingpedestrian right of way violations by vehicles is provided. In oneembodiment, the system combines two sensor modalities to monitor trafficintersections and track pedestrian movement and vehicle traffic. Thesystem identifies vehicles that violate pedestrian right of way andrecords and reports evidence of violations by the vehicles. For example,the system determines when pedestrians are legally within a crosswalkand are endangered by a vehicle, or when a vehicle is illegally stoppedwithin a crosswalk. Evidence will be collected in the form of a videosegment of the vehicle, still imagery of the driver and the licenseplate, the date and time, and the location, for example.

A system according to an exemplary embodiment comprises memory and logicconfigured to receive and store in the memory radar and video dataindicative of possible pedestrians and vehicles in an area underobservation. The logic segments and classifies the radar and video dataand stores in the memory tracked radar and video objects. The logic isfurther configured to receive and store in the memory traffic rules dataindicative of traffic laws for the area under observation. The logicprocesses the tracked radar and video objects with the traffic rulesdata to generate and store in the memory data indicative of pedestrianright of way violations.

A method according to an exemplary embodiment of the present disclosurecomprises receiving raw video data from a video camera and a radardevice collecting an intersection of interest; processing the raw videodata and radar data to form packetized video and radar data; segmentingthe video data and radar data and classifying objects of interest;tracking the radar and video objects of interest; processing trafficrules, and generating rules violations.

DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a block diagram illustrating a system in accordance with anexemplary embodiment of the present disclosure.

FIG. 2 is an exemplary video imaging sensor as depicted in FIG. 1.

FIG. 3 is an exemplary radar device as depicted in FIG. 1.

FIG. 4 is an exemplary sensor control device as depicted in FIG. 1.

FIG. 5 is a flowchart depicting exemplary architecture and functionalityof the rules logic in accordance with an exemplary embodiment of thedisclosure.

FIG. 6 is a video image of an intersection under observation with apedestrian in a crosswalk crossing the street and a vehicle approachingthe crosswalk.

FIG. 7 is a segmented video image of the intersection of FIG. 6.

FIG. 8 is an image of the intersection of FIG. 7 following applicationof a blob finder program.

FIG. 9 is the video image of FIG. 6 showing a target pedestrian and atarget vehicle with unique identification numbers assigned.

FIG. 10 is an exemplary illustration of packetized radar data collectedin an intersection under observation.

FIG. 11 is an exemplary illustration of the data from FIG. 10 aftersegmentation.

FIG. 12 is a flowchart depicting exemplary architecture andfunctionality of the correlation step in accordance with an exemplaryembodiment of the disclosure.

FIG. 13 is a flowchart depicting exemplary architecture andfunctionality of the tracking step in accordance with an exemplaryembodiment of the disclosure.

FIG. 14 illustrates coverage area on a surface area under observation ofa combined video imaging sensor and radar device.

FIG. 15 is an overhead representation of the system of FIG. 1, andspecifically of an intersection under observation by a combined sensor.

FIG. 16 depicts the system of FIG. 15 observing an intersection, withcombined sensors mounted facing each of the four directions north, west,south, and east and a vehicle driving north toward the intersection.

FIG. 17 illustrates the system of FIG. 16 as the vehicle passes throughthe intersection and is in range of the west combined sensor.

FIG. 18 illustrates the system of FIG. 16 as the vehicle continuesdriving north of the intersection.

FIG. 19 depicts the system of FIG. 15 observing an intersection, while anorthbound vehicle approaches the intersection and prepares to turn leftat the intersection.

FIG. 20 depicts the system of FIG. 19 as the vehicle is making the leftturn.

FIG. 21 depicts the system of FIG. 19 as the vehicle continues drivingwest.

FIG. 22 depicts the system of FIG. 15 observing an intersection, while anorthbound vehicle approaches the intersection and prepares to turnright at the intersection

FIG. 23 depicts the system of FIG. 22 as the vehicle is making the rightturn.

FIG. 24 depicts the system of FIG. 22 as the vehicle continues drivingeast.

FIG. 25 is a flowchart depicting exemplary architecture andfunctionality of the rules logic in accordance with an alternateexemplary embodiment of the disclosure in which only one sensor modalityis employed in the system.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 in accordance with an exemplaryembodiment of the present disclosure. The system 100 comprises a videoimaging sensor 101 and a radar device 102 which collect data, generallyof a traffic intersection (not shown). One or more pedestrians 103 maybe crossing a street (not shown) at the intersection and one or morevehicles 104 may be approaching the intersection. The system 100collects radar and video data to track vehicles 104 and pedestrians 103and determines when the vehicles 104 violate pedestrian right of way.

The video imaging sensor 101 comprises a video imaging device (notshown) such as a digital video camera that collects video images in itsfield of view. The video imaging sensor 101 further comprises a framegrabber (not shown) that packetizes video data and stores it, as furtherdiscussed herein with respect to FIG. 2.

The radar device 102 collects range, angle, and signal strength datareflected from objects in its field of view. The range, angle and signalstrength data are packetized within the radar device 102. The radardevice 102 is discussed further with respect to FIG. 3 herein.

The video imaging sensor 101 and the radar device 102 send packetizedvideo data (not shown) and packetized radar data (not shown) to a sensorcontrol device 107 over a network 105. The sensor control device 107 maybe any suitable computer known in the art or future-developed. Thesensor control device 107 may be located in a traffic box (not shown) atthe intersection under observation by the system 100, or may be locatedremotely. The sensor control device 107 receives the packetized videodata and radar data, segments the video data and radar data to classifyobjects of interest, tracks the objects of interest, and the processesrules to identify traffic violations. The sensor control device 107 isfurther discussed with respect to FIG. 4 herein.

In one embodiment, a user (not shown) accesses the sensor control device107 via a remote access device 106. Access to the remote access device106 may be made, for example, by logging into a website (not shown)hosted remotely, by logging in directly over a wireless interface, or bydirect connection via a user console (not shown). In one embodiment theremote access device 106 is a personal computer. In other embodiments,the remote access device 106 is a personal digital assistant (PDA),computer tablet device, laptop, portable computer, cellular or mobilephone, or the like. The remote access device 106 may be a computerlocated at, for example, the local police office (not shown).

The network 105 may be of any type network or networks known in the artor future-developed, such as the internet backbone, Ethernet, Wifi,WiMax, broadband over power line, coaxial cable, and the like. Thenetwork 105 may be any combination of hardware, software, or both.

FIG. 2 depicts an exemplary video imaging sensor 101 according to anembodiment of the present disclosure. The video imaging sensor 101generally comprises a frame grabber 110 and a video camera 128.

The frame grabber 110 comprises frame grabber logic 120, raw video data121, and packetized video data 122. In the exemplary video imagingsensor 101, frame grabber logic 120, raw video data 121 and packetizedvideo data 122 are shown as stored in memory 123. The frame grabberlogic 120, the raw video data 121, and the packetized video data 122 maybe implemented in hardware, software, or a combination of hardware andsoftware.

The frame grabber 110 captures raw video data 121 and packetizes it tofoul′ packetized video data 122. The packetized video data 122 is thensent to the sensor control device 107 (FIG. 1).

The frame grabber 110 also comprises a frame grabber processor 130,which comprises a digital processor or other type of circuitryconfigured to run the frame grabber logic 120 by processing andexecuting the instructions of the frame grabber logic 120. The framegrabber processor 130 communicates to and drives the other elementswithin the frame grabber 110 via a local interface 124, which caninclude one or more buses. A video network device 126, for example, auniversal serial bus (USB) port or other type network device connectsthe frame grabber 110 with the network 105 (FIG. 1) for communicationwith other network devices, such as the sensor control device 107(FIG. 1) and the remote access device 106 (FIG. 1).

When stored in memory 123, the frame grabber logic 120, the raw videodata 121 and the packetized video data 122 can be stored and transportedon any computer-readable medium for use by or in connection with aninstruction execution system, apparatus, or device, such as acomputer-based system, processor-containing system, or other system thatcan fetch the instructions from the instruction execution system,apparatus, or device and execute the instructions. In the context ofthis document, a “computer-readable medium” can be any means that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus, ordevice. The computer readable medium can be, for example but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, device, or propagation medium. Notethat the computer-readable medium could even be paper or anothersuitable medium upon which the program is printed, as the program can beelectronically captured, via for instance optical scanning of the paperor other medium, then compiled, interpreted or otherwise processed in asuitable manner if necessary, and then stored in a computer memory.

FIG. 3 depicts a radar device 102 according to an embodiment of thepresent disclosure. The exemplary radar device 102 generally comprises aradar sensor 310 and a radar network device 326. The radar device 102further comprises radar logic 320 and radar data 321, which can besoftware, hardware, or a combination thereof.

The radar sensor 310 comprises a radar transmitter and a receiver (notshown). The radar sensor 310 further comprises a digital processor orother type of circuitry configured to run the radar logic 320 byprocessing and executing the instructions of the radar logic 120. Theradar sensor 310 communicates to and drives the other elements withinthe radar device 102 via a local interface 324, which can include one ormore buses. A radar network device 326, for example, a universal serialbus (USB) port or other type network device connects the radar device102 with the network 105 (FIG. 1) for communication with other networkdevices, such as the sensor control device 107 (FIG. 1) and the remoteaccess device 106 (FIG. 1).

When stored in memory 323, the radar logic 320 and the radar data 321can be stored and transported on any computer-readable medium for use byor in connection with an instruction execution system, apparatus, ordevice, such as a computer-based system, processor-containing system, orother system that can fetch the instructions from the instructionexecution system, apparatus, or device and execute the instructions. Theradar data 321 comprises raw range data 330, raw angle data 331, and rawsignal strength data 332. Raw range data 330 comprises raw data receivedfrom the radar sensor 310 indicating the distance an object underobservation (not shown) is from the radar device 102. Raw angle data 331comprises data indicating the angle between the radar device 102 and theobject under observation. Raw signal strength data 332 comprises dataindicating the strength of the signal received from the object underobservation.

The radar logic 320 executes the process of receiving the raw range data330, raw angle data 331, and raw signal strength data 332 andpacketizing it to form packetized radar data 333. The packetized radardata 333 is sent to the sensor control device 107, as further discussedherein.

FIG. 4 depicts a sensor control device 106 according to an embodiment ofthe present disclosure. The sensor control device 106 generallycomprises a processing unit 171, a network device 176, an input device177, and optionally a display device 178.

The sensor control device 106 further comprises rules logic 174 andrules data 182 which can be software, hardware, or a combinationthereof. In the sensor control device 106, rules logic 174 and rulesdata 182 are shown as software stored in memory 423. However, the ruleslogic 174 and rules data 182 may be implemented in hardware, software,or a combination of hardware and software in other embodiments.

The processing unit 171 may be a digital processor or other type ofcircuitry configured to run the rules logic 174 by processing andexecuting the instructions of the rules logic 174. The processing unit171 communicates to and drives the other elements within the sensorcontrol device 106 via a local interface 175, which can include one ormore buses. Furthermore, the input device 177, for example, a keyboard,a switch, a mouse, and/or other type of interface, can be used to inputdata from a user (not shown) of the sensor control device 106, and thedisplay device 178 can be used to display data to the user. In addition,an network device 176, for example, a universal serial bus (USB) port orother type network device connects the sensor control device 106 withthe network 105 (FIG. 1) for communication with other network devices,such as the radar device 102, video imaging sensor 101, the trafficsignal 108, and the remote access device 106.

An exemplary input device 177 may include, but is not limited to, akeyboard device, switch, mouse, serial port, scanner, camera,microphone, or local access network connection. An exemplary displaydevice 178 may include, but is not limited to, a video display.

Exemplary rules data 182 comprises packetized radar data 333 receivedfrom the radar device 102 and packetized video data 122 received fromthe video imaging sensor 101. Exemplary rules data 182 may furthercomprise segmented video data 334, segmented radar data 335, classifiedvideo object data 336, classified radar object data 337, correlatedobject data 338, and track table data 339.

The rules data 182 further comprises traffic signal state data 173received from the traffic signal 108. In one embodiment, the trafficsignal 108 directly communicates its current state to the sensor controldevice 107 in the form of traffic signal state data 173 which indicateswhether the traffic signal 108 is red, yellow, or green, for example. Orfor a pedestrian traffic signal, the signal state data 173 may be“Walk,” “Don't Walk,” or a flashing “Don't Walk.” In another embodiment,the state of the traffic signal may be collected by the video imagingsensor 101, i.e., the video imaging sensor 101 can detect the color ofthe traffic light and report the state to the sensor control device 107.In still another embodiment, the intersection under observation does nothave traffic signals at all (e.g., a four way stop). The system 100therefore does not require input from traffic signals in order tomonitor an intersection and report certain violations.

The traffic rules data 181 comprises traffic rules and parameters forthe intersection under observation (not shown). Non-limiting examples oftraffic rules data 181 may include:

a. Vehicle Traffic Rules and Parameters:

-   -   i. Whether right turns are allowed on red;    -   ii. Whether traffic is one way or two-way;    -   iii. What constitutes a crosswalk obstruction, e.g., full        obstruction or partial obstruction?    -   iv. Is the area under observation an intersection, or a        mid-block crosswalk?    -   v. Is the traffic controlled by traffic signals? Pedestrian        signals? Stop signs?

b. Pedestrian Rules and Parameters:

-   -   i. Is a pedestrian allowed to cross on “Walk” indication only?    -   ii. Does a pedestrian always have the right of way?

c. System Rules and Parameters:

-   -   i. What is the proximity threshold (i.e., permitted distance        between pedestrian and vehicle)?    -   ii. Is a driver photo required as evidence of a violation?    -   iii. Is a video of the vehicle's approach required as evidence        of a violation?    -   iv. Does the video of the violation need to be a certain length,        or include certain objects, or angles, etc.?

The rules logic 174 executes the process of generating violation data180 by processing the traffic signal state data 173, the packetizedradar data 172, the packetized video data 183, and the traffic rulesdata 181. The violation data 180 can then be accessed by a user (notshown) via the remote access device 106.

The violation data 180 may include information such as a description ofthe violation, a description of the vehicle, a photo of the vehicle, aphoto of the license plate of the vehicle, a video of the violation, andthe like.

FIG. 5 is a flowchart depicting exemplary architecture and functionalityof the rules logic 174 (FIG. 4) in accordance with an exemplaryembodiment of the disclosure. In step 501, the sensor control device 107(FIG. 1) receives packetized video data 122 (FIG. 2) from the videoimaging sensor 101 (FIG. 1).

In step 502, the sensor control device 107 segments the packetized videodata 122 and creates segmented video data 334 (FIG. 4). In thissegmentation step 502, objects that are not part of the background of anarea under observation are segmented (i.e., subtracted from) from thebackground, and dynamic objects are observed. Any of a number of knownsegmentation algorithms may be used to perform this step. The segmentedvideo data 334 may include data such as: the height and width (inpixels) of an object under observation; the average color of the objectunder observation; the row and column position of the object underobservation, and the like. Each object under observation is assigned aunique identification number.

FIG. 6 illustrates an intersection under observation 600 with a “target”pedestrian 601 in a crosswalk 603 crossing the street and a targetvehicle 602 approaching the crosswalk 602. FIG. 6 is an exemplary frameof video data 122 before the segmentation step 502 has been performed.FIG. 7 is an exemplary frame of segmented video data. The staticbackground is black and the target pedestrian 601 and the target vehicle602 show up as whitish “blobs.”

In FIG. 8, a “blob finder” program has been applied to the segmentedvideo data frame. The blob finder program finds all of the pixels inproximity and groups them and illustrates the object in a specifiedcolor. At this point, the pedestrian 601 appears with more clarity as alight blue blob and the vehicle 602 appears as a red blob. FIG. 9illustrates a video image of the intersection of FIGS. 6-8, with theunique identification numbers assigned in step 502 and a green trackingbox surrounding the pedestrian 601 and vehicle 602.

Referring to FIG. 5, in step 503, the sensor control device 107classifies the segmented video data 334. In this classification step503, the sensor control device 107 analyzes the blobs from step 502 anddecides what they are: e.g., whether they are pedestrians, vehicles, orunknown. The sensor control device thus classifies the segmented videodata 334, resulting in classified video objects 336.

In parallel with steps 701-703, the frame grabber logic 120 in step 504receives packetized radar data 333 (FIG. 3) from the radar device 102(FIG. 1). In step 505, the packetized radar data 333 (illustrated inFIG. 10) is segmented in a similar manner as the segmentation of thevideo data in step 502 (and as further discussed herein with respect toFIG. 11), resulting in segmented radar data 335.

FIG. 10 illustrates packetized radar data 333 received from the radardevice 102 (FIG. 1) from an observation of traffic at an intersectionover time. Approaching vehicles create tracks 800, i.e. lines ofindividual radar signals received over time. Where the tracks becomehorizontal (indicated by reference number 801), the vehicle has stopped,i.e., as in at a red light in the intersection. In the segmentation step505, the radar data is decluttered, meaning that the background objects(e.g., the lines indicated by reference numbers 802 and 803) areidentified and removed.

FIG. 11 illustrates segmented radar data 335 after the segmentation step505 has been completed. Reference number 805 indicates a vehicle thathas passed through the intersection without stopping. Reference number806 indicates a vehicle that approached the intersection, stopped at ared light, and then passed on through the intersection. Reference number807 indicates a pedestrian that has crossed through the intersection atthe crosswalk.

In step 506, the radar objects under observation are classified in asimilar manner as the classification of the video data in step 503. Inthis regard, from the segmented radar data 335, radar objects underobservation are classified into vehicles or pedestrians or unknowns, andidentification numbers are assigned to the objects, resulting inclassified radar objects 337.

In step 507, the classified video objects 336 from step 503 and theclassified radar objects 337 from step 506 are correlated. The goal ofthis correlation step 507 is to take the classified objects from eachsensor modality and create a consolidated list of well defined objects(or correlated objects 338) to the tracker. The more information isknown about each object, the better the results of the tracking step 508will be. For example, from the classified video objects 336, the systemcan determine the color of a vehicle, the angle from the camera to thevehicle, the number of pixels the vehicle fills in the camera view, andthe like, at an instant in time. From the classified radar objects 337,the system can determine the range, speed, and angle of the vehicle, atan instant in time. By monitoring multiple frames of video and radar,the system can compute a velocity for the vehicle and rate of change ofthe vehicle speed.

In the correlation step 507, like parameters for radar and video objectsare compared and correlated. For example, a blue car observed at anangle of −4 degrees via the video data can be correlated with a vehicleseen at −4 degrees via the radar data, and a correlated object 338 isrecorded for tracking. A confidence level is assigned to each correlatedobject 338 based upon the likelihood of correlation of the twomodalities based upon the observed parameters. For example, where aclassified object has the same angle value and range value and speedvalue as a radar-observed object, a high degree of confidence would beassigned. However, if for one or more of the observed parameters, thevideo data shows something the radar data does not, such that theobjects are not well correlated, the confidence value would be lower.The correlation step 507 is further discussed herein with respect toFIG. 12.

In step 508, the correlated objects 338 are tracked. In this step, atracking algorithm that is known in the art filters the objects toidentify vehicles and pedestrians in the area under observation. Any ofa number of known tracking algorithms may be used for the tracking step508, such as a particle filtering or Kalman filtering. An exemplarytracking step 508 is further discussed herein with respect to FIG. 13.

In step 509, the rules logic 174 (FIG. 4) executes the process ofgenerating violation data 180 (FIG. 4) by applying the traffic rules tothe movement of the vehicles and pedestrians. By way of example, ifvehicles are supposed to stop at a crosswalk when pedestrians arepresent, and one of these vehicles is tracked passing into the crosswalk(i.e., not stopping) when a pedestrian is present, a violation hasoccurred. The violation data 180 generated includes the trafficviolation that occurred, time and date data of the violation, andphotographs of the vehicle that had the violation, and a video of theviolation. A traffic official can access the violation data 180 via theremote access device 106 and has all of the evidence needed to reportthe violation and issue a citation.

FIG. 12 is a flowchart depicting exemplary architecture andfunctionality of the correlation step 507 (FIG. 5) in accordance with anexemplary embodiment of the disclosure. In step 1201, the rules logic174 (FIG. 4) searches classified video objects 336 for correlatingclassified radar objects 337. In step 1202, for various properties ofeach modality, record in a matrix the “like” measurements. In oneexemplary embodiment, the matrix may be an M×N×Z matrix where “M” is thenumber of objects in one modality (e.g., video), “N” is the number ofobjects in the alternate modality (e.g., radar), and “Z” is the numberof related properties (e.g., angle, speed, etc.).

In step 1203, an overall “likeness” score is computed for eachcorrelated object 338 in the two modalities. In the exemplary objectdiscussed above with respect to step 1202, the computation uses the Zvector and the result will be an M×N matrix. In step 1204, a confidencevalue is assigned for each object based upon the likeness scores fromstep 1203.

FIG. 13 is a flowchart depicting exemplary architecture andfunctionality of the tracking step 508 (FIG. 5) in accordance with anexemplary embodiment of the disclosure. In step 1301, the rules logic174 predicts state vectors for prior known correlated objects 338. Inthis step, the system uses past object data to predict currentparameters of the object, and creates predicted state vectors. Statevector data is recorded as track table data 339.

In step 1302, the state vectors from step 1301 are updated based uponcurrent observed objects. In other words, this step determines howaccurate the predicted state vectors were for the observed objects. Instep 1303, state vectors are added for “new” objects. The new objectsare objects that did not line up with state vector predictions, suchthat they may not be pedestrians or vehicles of interest.

In step 1304, the state vectors are trimmed, and “stale” objects arediscarded. For example, if an object has not been seen in three or foursets of data over time, it is not a pedestrian or vehicle of interestand can be removed from the track list in the track table data 339.

The system 100 described herein with respect to FIG. 1-FIG. 12 generallydescribes a single radar device 102 and a single video imaging sensor101 for the sake of simplicity. Obviously, however, an intersectionunder observation generally requires more than one radar device 102 andvideo imaging sensor 101, to observe all of the traffic lanes and recordthe necessary information for reporting violations, as is furtherdiscussed herein.

FIG. 14 illustrates coverage area on a surface area under observation 53of a combined video imaging sensor 101 and radar device 102 (FIG. 1),which will be referred to as the “combined sensor” 50 with reference toFIGS. 14-24 herein. The combined sensor 50 provides a relative longradar coverage area 51 and a shorter, but wider, video coverage area 52.The radar coverage area 51 being longer provides input for trackingvehicles (not shown) from farther away, as the vehicles approach anintersection.

FIG. 15 is an overhead representation of the system 100 (FIG. 1), andspecifically of an intersection 55 under observation by a combinedsensor 50. The combined sensor 50 is typically mounted to an overheadpole or wire (not shown), such that the combined sensor 50 overlooks theintersection 55 from above.

The intersection 55 comprises a north-south street 60 intersecting withan east-west street 61. The intersection 55 further comprises fourpedestrian crosswalks: a north crosswalk 56, an east crosswalk 57, asouth crosswalk 58, and a west crosswalk 59. FIG. 15 is a simplerepresentation showing one combined sensor 50 with a radar coverage area51 and a video coverage area 52. It is understood, however, that forcomplete coverage of a four-street intersection, four (4) combinedsensors 50, one facing in each of the four street directions, wouldgenerally be required, as further discussed herein. Further, it isunderstood that intersections of different configurations (e.g., Yintersections, five point intersections, mid-block crosswalks, and thelike) would generally require a different number of combined sensors.

FIG. 16 depicts the system 100 of FIG. 15 observing an intersection 55,with combined sensors 50 a-50 d mounted facing each of the fourdirections north, west, south, and east, respectively. In operation ofthe system 100, most pedestrian right of way violations fall into one ofthe following four broad categories based upon the traffic path of avehicle 63:

a. Normal through traffic;

b. Left turn;

c. Right turn;

d. Right turn on red.

Each category of violation requires proper coordination between thecombined sensors 50 a-50 d to properly track the intersection 55. Thefour categories of violations are discussed in Examples 1-4 below.Although input from the traffic signal 108 (FIG. 1) is not generallydiscussed in Examples 1-4 below, it is understood that traffic signalstate data 173 (FIG. 4) may be collected and analyzed in any or all ofthe scenarios.

Example 1: Normal Straight Through Traffic Scenario Example

FIGS. 16-18 illustrate an exemplary traffic scenario for a northboundvehicle 63 traveling straight through the intersection 55 (i.e., notturning right or left). In this scenario, examples of possibleviolations are:

a. an illegal stop by the vehicle 63 within the north crosswalk 56; and

b. an illegal stop within the south crosswalk 58.

In this scenario, unless stopped by a traffic signal (not shown), thevehicle 63 generally has the right of way to proceed north through thesouth crosswalk 58. Under normal circumstances there are no moving rightof way violations that could occur where a pedestrian's safety isillegal endangered. However, if the vehicle 63 becomes stopped on eitherthe north crosswalk 56 or the south crosswalk 58 at the conclusion of agreen light, then a violation has occurred that should be cited.

In the scenario illustrated in FIG. 16, the vehicle 63 is driving northon street 60, south of the intersection 55. The vehicle 63 is firstdetected by the radar device in the north combined sensor 50 a, and thenby the video imaging sensor in the north combined sensor 50 a. The northcombined sensor 50 a collects and stores a sequence of video and radarframes of the approach of the vehicle 63.

FIG. 17 illustrates the vehicle 63 as it passes through the intersection55 and is in range of the west combined sensor 50 b. A still image ofthe driver (not shown) is collected by the west combined sensor 50 b andstored as raw video data 121 (FIG. 2).

FIG. 18 illustrates the vehicle 63 as it continues driving north onstreet 60, north of the intersection 55. At this point, the vehicle 63is in range of the south combined sensor 50 c, which collects and storesa sequence of video frames of the vehicle 63 passing through the northcrosswalk 56, and collects and stores a still image of the vehicle'slicense plate, as well as the date and time.

In the traffic sequence discussed above with respect to FIGS. 16-18, ifthe sensor control device 107 (FIG. 1) detects that the vehicle 63stopped in a crosswalk 56, 57, 58, or 59, then the data recorded of thevehicle 63 would be tagged as evidence and stored as violation data 180(FIG. 4). However, if the sensor control device 107 detects that noviolation has occurred, the data associated with vehicle 63 would bediscarded.

Example 2: Left Turn Scenario Example

FIGS. 19-21 illustrate an exemplary traffic scenario for a northboundvehicle 63 turning left at the intersection 55. In this scenario,examples of possible violations are:

-   -   a. an illegal stop by the vehicle 63 within the south crosswalk        58; and    -   b. an illegal moving within the west crosswalk 58 when a        pedestrian is in the crosswalk.

Under the left turn scenario, the vehicle 63 has the right of way toproceed through the south crosswalk 58, but must yield to pedestrians inthe west crosswalk 59.

In the scenario illustrated in FIG. 19, the vehicle 63 is driving northon street 60, south of the intersection 55. The vehicle 63 is firstdetected by the radar device in the north combined sensor 50 a, and thenby the video imaging sensor in the north combined sensor 50 a. The northcombined sensor 50 a collects and stores a sequence of video and radarframes of the approach of the vehicle 63 and of the vehicle passingthrough the south crosswalk 58.

FIG. 20 illustrates the vehicle 63 as it turns left in the intersection55 and is in range of the west combined sensor 50 b. A still image ofthe driver (not shown) is collected by the west combined sensor 50 b andstored as raw video data 121 (FIG. 2).

FIG. 21 illustrates the vehicle 63 as it continues driving west throughthe intersection on street 61. At this point, the vehicle 63 is in rangeof the east combined sensor 50 d, which collects and stores a sequenceof video frames of the vehicle 63 passing through the west crosswalk 59,and collects and stores a still image of the vehicle's license plate, aswell as the date and time.

In the traffic sequence discussed above with respect to FIGS. 19-21, ifthe sensor control device 107 (FIG. 1) detects that the vehicle 63stopped in the south crosswalk 58 then the data recorded of the vehicle63 would be tagged as evidence and stored as violation data 180 (FIG.4). Further, if a proximity violation occurs in the west crosswalk 59(i.e., the vehicle 63 gets too close to a pedestrian (not shown) in thecrosswalk 59), then the data recorded of the vehicle 63 would be taggedas evidence and stored as violation data 180 (FIG. 4). However, if thesensor control device 107 detects that no violation has occurred, thedata associated with the vehicle 63 would be discarded.

Example 3: Right Turn Scenario Example

FIGS. 22-24 illustrate an exemplary traffic scenario for a northboundvehicle 63 turning right at the intersection 55. In this scenario,examples of possible violations are:

a. an illegal stop by the vehicle 63 within the south crosswalk 58; and

b. illegal moving within the east crosswalk 58 when a pedestrian is inthe crosswalk.

Under the right turn scenario, the vehicle 63 has the right of way toproceed through the south crosswalk, but must yield to pedestrians (notshown) in the east crosswalk.

In the scenario illustrated in FIG. 22, the vehicle 63 is driving northon street 60, south of the intersection 55. The vehicle 63 is firstdetected by the radar device in the north combined sensor 50 a, and thenby the video imaging sensor in the north combined sensor 50 a. The northcombined sensor 50 a collects and stores a sequence of video and radarframes of the approach of the vehicle 63 and of the vehicle passingthrough the south crosswalk 58.

FIG. 23 illustrates the vehicle 63 as it turns right in the intersection55 and is in range of the west combined sensor 50 b. A still image ofthe driver (not shown) is collected by the west combined sensor 50 b andstored as raw video data 121 (FIG. 2).

FIG. 24 illustrates the vehicle 63 as it continues driving east throughthe intersection 55. At this point, the vehicle 63 is still in range ofthe west combined sensor 50 b, which collects and stores a sequence ofvideo frames of the vehicle 63 passing through the east crosswalk 57,and collects and stores a still image of the vehicle's license plate, aswell as the date and time.

In the traffic sequence discussed above with respect to FIGS. 22-24, ifthe sensor control device 107 (FIG. 1) detects that the vehicle 63stopped in the south crosswalk 58 then the data recorded of the vehicle63 would be tagged as evidence and stored as violation data 180 (FIG.4). Further, if a proximity violation occurs in the east crosswalk 57(i.e., the vehicle 63 gets too close to a pedestrian (not shown) in thecrosswalk 57), then the data recorded of the vehicle 63 would be taggedas evidence and stored as violation data 180 (FIG. 4). However, if thesensor control device 107 detects that no violation has occurred, thedata associated with the vehicle 63 would be discarded.

Example 4: Right Turn on Red Scenario Example

FIGS. 22-24 also can be used to illustrate an exemplary traffic scenariofor a northbound vehicle 63 turning right on red at the intersection 55.In this scenario, examples of possible violations are:

a. illegal moving within the south crosswalk 58 when a pedestrian is inthe crosswalk;

b. an illegal stop by the vehicle 63 within the south crosswalk 58; and

c. an illegal stop by the vehicle 63 within the east crosswalk 57.

Under the right turn on red scenario, the pedestrian has the right ofway to proceed through the south crosswalk 58, and the vehicle 63 mustyield to pedestrians, if present. In addition, the vehicle 63 must notstop in the south crosswalk 58 while the pedestrians have the right ofway, or in the east crosswalk at the change of traffic signal, thusblocking pedestrian access to that crosswalk.

In the scenario illustrated in FIG. 22, the vehicle 63 is driving northon street 60, south of the intersection 55. The vehicle 63 is firstdetected by the radar device in the north combined sensor 50 a, and thenby the video imaging sensor in the north combined sensor 50 a. The northcombined sensor 50 a collects and stores a sequence of video and radarframes of the approach of the vehicle 63 and of the vehicle passingthrough the south crosswalk 58.

FIG. 23 illustrates the vehicle 63 as it turns right in the intersection55 and is in range of the west combined sensor 50 b. A still image ofthe driver (not shown) is collected by the west combined sensor 50 b andstored as raw video data 121 (FIG. 2).

FIG. 24 illustrates the vehicle 63 as it continues driving east throughthe intersection 55. At this point, the vehicle 63 is still in range ofthe west combined sensor 50 b, which collects and stores a sequence ofvideo frames of the vehicle 63 passing through the east crosswalk 57,and collects and stores a still image of the vehicle's license plate, aswell as the date and time.

In the traffic sequence discussed above with respect to FIGS. 22-24, ifthe sensor control device 107 (FIG. 1) detects that the vehicle 63stopped in the south crosswalk 58 then the data recorded of the vehicle63 would be tagged as evidence and stored as violation data 180 (FIG.4). Further, if a proximity violation occurs in the east crosswalk 57 orif the vehicle 63 were stopped in the east crosswalk 57 when the trafficsignal (not shown) changes, then the data recorded of the vehicle 63would be tagged as evidence and stored as violation data 180 (FIG. 4).However, if the sensor control device 107 detects that no violation hasoccurred, the data associated with the vehicle 63 would be discarded.

FIG. 25 is a flowchart depicting exemplary architecture andfunctionality of the rules logic 174 (FIG. 4) in accordance with analternate exemplary embodiment of the disclosure. In this embodiment,the system 100 (FIG. 1) does not utilize the radar device 102, andinstead of two sensor modalities, relies on video to track vehicles.

In step 2501, the sensor control device 107 (FIG. 1) receives packetizedvideo data 122 (FIG. 2) from the video imaging sensor 101 (FIG. 1).

In step 2502, the sensor control device 107 segments the packetizedvideo data 122 and creates segmented video data 334 (FIG. 4), in thesame manner as discussed above with respect to FIGS. 5-8. In step 2503,the sensor control device 107 classifies the segmented video data 334 ina manner similar to that discussed above with respect to FIG. 5.

In step 2504, the segmented video objects are tracked. In this step, atracking algorithm that is known in the art filters the objects toidentify vehicles and pedestrians in the area under observation. Thetracking step 2504 is similar to step 508 discussed with respect to FIG.5 herein.

In step 2505, the rules logic 174 (FIG. 4) executes the process ofgenerating violation data 180 (FIG. 4) by applying the traffic rules tothe movement of the vehicles and pedestrians, as discussed above withrespect to step 509 of FIG. 5.

This disclosure may be provided in other specific forms and embodimentswithout departing from the essential characteristics as describedherein. The embodiments described are to be considered in all aspects asillustrative only and not restrictive in any manner.

What is claimed is:
 1. A system, comprising: a video imaging sensorconfigured to record video data indicative of possible pedestrians andvehicles in an area under observation; a radar device configured torecord radar data indicative of possible pedestrians and vehicles in anarea under observation; and a sensor control device configured tosegment and classify the radar and video data and correlate classifiedradar objects with classified video objects to identify correlatedobjects of interest, the sensor control device further configured totrack the correlated objects to identify tracked pedestrians andvehicles, the sensor control device further configured to receive andstore in the memory traffic rules data indicative of traffic laws forthe area under observation, the sensor control device further configuredto process the tracked pedestrians and vehicles with the traffic rulesdata to generate and store in the memory data indicative of allegedtraffic violations.
 2. The system of claim 1, wherein sensor controldevice configured to correlate classified radar objects with classifiedvideo objects comprises logic configured to search classified videoobjects for correlating classified radar objects, record a likeness ofthe modalities for a property, and determine a likelihood of correlationbetween the modalities.
 3. The system of claim 2, wherein the logicconfigured to determine the likelihood of correlation between themodalities is further configured to compute a temporal likeness scorefor each correlated object.
 4. The system of claim 3, wherein the logicconfigured to determine the likelihood of correlation of the modalitiesis further configured to assign a confidence value for each object basedupon the temporal likeness score.
 5. The system of claim 1, wherein thesensor control device is further configured to receive and store inmemory traffic signal state data indicative of the state of a vehicletraffic signal corresponding to activity of the tracked radar and videoobjects.
 6. The system of claim 5, the traffic signal state data furthercomprises data indicative of the state of a pedestrian crosswalk signal.7. The system of claim 1, wherein the area under observation includesone or more pedestrian crosswalks.
 8. A method of recording evidence ofsuspected traffic violations, the method comprising: receiving raw videodata from one or more video cameras collecting video data of an areaunder observation; processing the raw video data to form packetizedvideo data; spatially segmenting the packetized video data andclassifying video objects of interest; receiving raw radar data from oneor more radar sensors collecting radar data of the area underobservation; spatially segmenting the raw radar data and classifyingradar objects of interest; correlating the radar and video objects ofinterest, tracking the correlated objects of interest; receiving trafficrules data indicative of traffic laws; applying the traffic rules datato the correlated objects of interest and recording alleged violationdata.
 9. The method of claim 8, wherein the step of correlating theradar and video objects of interest further comprises searchingclassified video objects for correlating classified radar objects,recording a likeness of the modalities for a property, and determining alikelihood of correlation between the modalities.
 10. The method ofclaim 9 wherein the correlating step comprising determining a likelihoodof correlation between the modalities further comprises computing atemporal likeness score for each correlated object.
 11. The method ofclaim 9, wherein the correlating step comprising determining alikelihood of correlation between the modalities further comprisesassigning a confidence value for each object based upon the temporallikeness score.
 12. The method of claim 8, further comprising receivingtraffic signal state data from a vehicle traffic signal and processingthe traffic signal state data with the traffic rules to record allegedviolation data.
 13. The method of claim 12, wherein the traffic signalstate data comprises data indicative of the state of a vehicle trafficsignal.
 14. The method of claim 13, wherein the traffic signal statedata further comprises data indicative of the state of a pedestriancrosswalk signal.
 15. The method of claim 8, further comprising remotelyaccessing the alleged violation data and generating traffic citations.16. A method of recording evidence of suspected traffic violations byvehicles, the method comprising: receiving raw video data from one ormore video cameras collecting video data of an area under observation;processing the raw video data to form packetized video data; spatiallysegmenting the packetized video data and classifying video objects ofinterest; receiving raw radar data from one or more radar sensorscollecting radar data of the area under observation; spatiallysegmenting the raw radar data and classifying radar objects of interest;correlating the radar and video objects of interest, tracking thecorrelated objects of interest; receiving traffic rules data indicativeof traffic laws; and applying the traffic rules data to the correlatedobjects of interest and recording alleged violation data.
 17. The methodof claim 16, wherein the tracking step comprises predicting positiondata of prior known objects, updating position data for prior knownobjects based upon current observed objects, adding position data fornew objects, and discarding position data for stale objects.
 18. Themethod of claim 16, further comprising remotely accessing the allegedviolation data and generating traffic citations.
 19. The method of claim16, wherein the tracking step of predicting position data of prior knownobjects comprises predicting state vectors of prior observed objects.20. The method of claim 19, wherein the tracking step of updatingposition data for prior known objects based upon current observedobjects comprises updating state vectors for prior known objects basedupon current state vectors.
 21. The method of claim 20, wherein thetracking step of adding position data for new objects comprises addingstate vectors for new objects.
 22. The method of claim 21, wherein thetracking step of discarding position data for stale objects comprisesdiscarding state vectors for objects that have not been observed over aplurality of sets of data over time.